Search Results: "Matthew Palmer"

25 December 2011

Matthew Palmer: The Other Way...

The profusion of network cables strung through doorways here demonstrates that two drops per sysadmin isn t anywhere near enough.

What I actually suspect it demonstrates is that Chris company hasn t learnt about the magic that is VLANs. All of the reasons he cites in the longer, explanatory blog post could be solved with VLANs. The only time you can t get away with one gigabit drop per office and an 8 port VLAN-capable switch is when you need high capacity, and given how many companies struggle by with wifi, I m going to guess that sustained gigabit-per-machine is not a common requirement. So, for Christmas, buy your colleages a bunch of gigabit VLAN capable switches, and you can avoid both the nightmare of not having enough network ports, and the more hideous tragedy of having to crawl around the roofspace and recable an entire office.

17 December 2011

Matthew Palmer: Rethtool: How I Learned to Stop Worrying and Love the ioctl

Damn those unshaven yaks I m trying to write a Nagios plugin for work that will comprehensively monitor network interfaces and make sure they re up, passing traffic, all those sorts of things. Of course, I m doing it all in Ruby, because that s how I roll. So, I need to Know Things about the interface. Everyone does that with ethtool. Right? Sure, if your eyeballs are parsing it. But have you ever tried to machine parse it? To put it as eloquently as possible:

# ethtool eth0
Settings for eth0:
 Supported ports: [ TP MII ]
 Supported link modes:   10baseT/Half 10baseT/Full 
                         100baseT/Half 100baseT/Full 
                         1000baseT/Half 1000baseT/Full 
 Supports auto-negotiation: Yes
 Advertised link modes:  10baseT/Half 10baseT/Full 
                         100baseT/Half 100baseT/Full 
                         1000baseT/Half 1000baseT/Full 
 Advertised pause frame use: No
 Advertised auto-negotiation: Yes
 Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                      100baseT/Half 100baseT/Full 
                                      1000baseT/Half 1000baseT/Full 
 Link partner advertised pause frame use: No
 Link partner advertised auto-negotiation: Yes
 Speed: 1000Mb/s
 Duplex: Full
 Port: MII
 PHYAD: 0
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: pumbg
 Wake-on: g
 Current message level: 0x00000033 (51)
 Link detected: yes

Parse that, bitch!

Or perhaps not. At any rate, I decided that it would be most advantageous if I went straight to the source and twiddle the ioctl until it did my bidding. And thus, about 5 hours later, was Rethtool born. Once I worked out a less-than-entirely-crackful way of dealing with C structs in Ruby (after a bit of digging around, I went with the appallingly-undocumented-but-sufficiently-featureful CStruct), and after I finally worked out I was passing the wrong damned struct to ioctl(SIOCETHTOOL) (speaking of appallingly-undocumented: fuck you, ioctl, and all your twisty-passages children), it was smooth sailing. So, if you re one of the eight or so people on earth who will ever need to get at the grubby internals of your network interfaces using Ruby (and can t do it via some sysfs magic), Rethtool is for you.

12 November 2011

Matthew Palmer: Misleading error messages from blktrace

If you ever get an error message from the blktrace tool that looks like this:

BLKTRACESETUP(2) /dev/dm-0 failed: 2/No such file or directory
Thread 3 failed open /sys/kernel/debug/block/(null)/trace3: 2/No such file or directory
Thread 2 failed open /sys/kernel/debug/block/(null)/trace2: 2/No such file or directory
Thread 0 failed open /sys/kernel/debug/block/(null)/trace0: 2/No such file or directory
Thread 1 failed open /sys/kernel/debug/block/(null)/trace1: 2/No such file or directory
FAILED to start thread on CPU 0: 1/Operation not permitted
FAILED to start thread on CPU 1: 1/Operation not permitted
FAILED to start thread on CPU 2: 1/Operation not permitted
FAILED to start thread on CPU 3: 1/Operation not permitted

Don t be alarmed your disk hasn t suddenly disappeared out from underneath you. In fact, it means quite the opposite of what No such file or directory might imply. In fact, it means that there is already a blktrace of that particular block device in progress, and you ll need to kill that one off before you can start another one. Thank $DEITY for the kernel source code it was the only hope I had of diagnosing this particular nit before I went completely bananas and smashed my keyboard into small pieces.

28 October 2011

Matthew Palmer: rsync for LVM-managed block devices

If you ve ever had to migrate a service to a new machine, you ve probably found rsync to be a godsend. It s ability to pre-sync most data while the service is still running, then perform the much quicker sync the new changes action after the service has been taken down is fantastic. For a long time, I ve wanted a similar tool for block devices. I ve managed ridiculous numbers of VMs in my time, almost all stored in LVM logical volumes, and migrating them between machines is a downtime hassle. You need to shutdown the VM, do a massive dd netcat, and then bring the machine back up. For a large disk, even over a fast local network, this can be quite an extended period of downtime. The naive implementation of a tool that was capable of doing a block-device rsync would be to checksum the contents of the device, possibly in blocks, and transfer only the blocks that have changed. Unfortunately, as network speeds approach disk I/O speeds, this becomes a pointless operation. Scanning 200GB of data and checksumming it still takes a fair amount of time in fact, it s often nearly as quick to just send all the data as it is to checksum it and then send the differences.¹ No, a different approach is needed for block devices. We need something that keeps track of the blocks on disk that have changed since our initial sync, so that we can just transfer those changed blocks. As it turns out, keeping track of changed blocks is exactly what LVM snapshots do. They actually keep a copy of what was in the blocks before it changed, but we re not interested in that so much. No, what we want is the list of changed blocks, which is stored in a hash table on disk. All that was missing was a tool that read this hash table to get the list of blocks that had changed, then sent them over a network to another program that was listening for the changes and could write them into the right places on the destination. That tool now exists, and is called lvmsync. It is a slightly crufty chunk of ruby that, when given a local LV and a remote machine and block device, reads the snapshot metadata and transfers the changed blocks over an SSH connection it sets up. Be warned: at present, it s a pretty raw piece of code. It does nothing but the send updated blocks over the network , so you have to deal with the snapshot creation, initial sync, and so on. As time goes on, I m hoping to polish it and turn it into something Very Awesome. Patches Accepted , as the saying goes.

rsync avoids a full-disk checksum because it cheats and uses file metadata (the last-modified time, or mtime of a file) to choose which files can be ignored. No such metadata is available for block devices (in the general case).

23 August 2011

Matthew Palmer: UPSes in Datacentres

(This was going to be a comment on this blog post, but it s a Turdpress site that wants JS and cookies to comment. Bugger that for a game of skittles. Rimuhosting s recent extended outage due to power problems was apparently caused by a transfer switch failure at their colo provider. This has led people to wonder if putting UPSes in individual racks is a wise move. The theory is that in the event of a small outage, the UPS can keep things humming, and in an extended outage you can gracefully shut things down rather than having a hard thump. I happen to think this theory is bunkum. Your UPS is a newly instituted single point of failure. I d be willing to bet that the cost of purchasing, installing, and maintaining the UPSes, as well as the cost of the outages that inevitably result from their occasional failure, would be far greater than the cost of the occasional power outage you get in a well-managed facility. Good facilities don t have small outages. They don t have squirrels in the roof cavities, and they don t have people dropping spanners across busbars. The only outages they have are the big ones, when some piece of overengineered equipment turns out to be not so overengineered the multi-hour (or multi-day) ones where your UPS isn t going to stop you from going down. Your SLA credit and customer goodwill is already toast, so all you re saving is the incremental cost of a little bit more downtime while you get fscks run. If you want the best possible power reliability, get yourself into a really well engineered facility, and run dual-power on everything. Definitely run the numbers before you go down the UPS road; I ll bet you find they re not worth it.

Matthew Palmer: Oh HP, you Bucket of Fail

I recently got given a new printer, a HP LaserJet Professional ¹ P1102w. It s fairly loudly touted on HP s website that this printer has Full support under Linux. And yet, it won t work with my Linux-based print server. Why? Because it uses a proprietary driver plugin, and that plugin is only available for x86 and amd64, and my print server is ARM-based. Well done, HP. You ve managed to revive the old all the world s a VAX philosophy, on an OS that is more than capable of running on practically anything. You got that for free. Why do you insist on screwing with it? As an added bonus, when I try to Ask a Question on the HPLIP website, to politely (ha!) inquire as to the possibility of an ARM binary, I get sent to Launchpad, which does nothing more than tell me that there is an Invalid OpenID transaction . That s the entire content of the page. Useful. Lies, damned lies, and a double helping of proprietary software fail. My day is complete.

I use scarequotes around Professional because, as far as I can tell, this is just an entry-level personal laser printer. There is nothing particularly professional about it.

21 August 2011

Matthew Palmer: Unintended Consequences: Why Evidence Matters

If you were trying to get rid of hiring discrimination (on grounds irrelevant to the ability to do the job), you d think a good way to do it would be to reduce the ability of the hiring manager to discriminate, by restricting their access to irrelevant (but possibly prejudicial) information. It s certainly what I might come up with as an early idea in a brainstorming session. I m not alone: France had this same idea, and gave it a go, by passing a law requiring companies to anonymise resumes before they got to any decision makers. So far, so average. But rather than just coming up with an idea and inflicting it on everyone by a blanket law, they did what should be done with all new ideas: they trialled it (with 50 large corporations, according to the report) before making it universal, to make sure that the theory matched reality. Then, after giving it a good shake, they examined the evidence, and found that the idea had some unintended consequences:

Applicants with foreign names, or who lived in under privileged areas were found to be less likely to be called in for an interview without the listing of their name and address. Researchers reasoned that this was because employers and recruiters made allowances for subpar presentation or limited French speaking if their performance could be explained by deprivation or foreign birth.

The icing on the cake is that now the evidence is in, they re now planning on making it optional (I m not sure how that s different from killing it entirely, but I guess it s worth the same in the end). So we ve got the quinella of decision-making awesome:

An idea was had
A trial was run
The evidence was examined
When the evidence didn t support the idea, the idea was abandoned

Far too often, we get far too attached to our ideas, and don t let them go when reality doesn t fit our preconceptions. Kudos to the people involved in this idea for not letting their egos get in the way of good government. Let it be an object lesson for us all.

19 August 2011

Matthew Palmer: Stream of Consciousness

This forum post on requiring formal letters of resignation made me smile:

HR does silly stuff like this all the time. Somebody s following some policy that was created because somebody verbally resigned nine years ago and then wanted to come back and some executive said where s their letter and HR said we don t have one and the exec said that s not good and we oughta not be doing stuff to help people leave unless they re really leaving and HR said okay we ll have a policy and the exec said that s good. And the exec s not there anymore.

I ll leave everyone to make their own conclusions as to why I was reading that particular thread.

22 July 2011

Matthew Palmer: Aaah, That Takes Me Back...

The Wrong Answer story from these tales from the interview brings back some memories I ve written this code (well, I got halfway through it before I realised precisely how stupid an idea it was), and I ve counselled more than one developer against trying it again (always unsuccessfully, much to my chagrin and that of anyone who had to deal with it in the future, I m sure). The reason people do it is twofold:

It s more fun to write this sort of code than the sort that actually solves a real problem; and
Schema manipulation in SQL databases sucks rocks. There s usually limited (or at least painful) introspection capabilities, and manipulating the schema is annoying and usually is deeply balls, performance wise.

If the resources that have been spent building these databases-on-databases over the years were instead redirected into fixing problem 2, above, then it probably wouldn t be a problem any more. On the other hand, problem 1 would still exist, for which the only solution is regular beatings. Remember: always whack your programmers on the back of the head at least once a week. Even if you don t know why, they do.

19 July 2011

Matthew Palmer: Static Comments in Jekyll

When I recently switched my blog over to Jekyll as my blog/site generation platform, one of the driving forces was implementing comments. I ve been a comment-free blogger for years, but I ve just decided to change my opinions on things and see what happens. (Turns out: you get some cool comments. And a lot of spam.) However, the standard advice when it comes to handling comments with Jekyll is use Disqus (or, very occasionally, some other Insert-Javascript-Here cloud commenting service). Not being one to stand for the conventional wisdom, and given that I m allergic to the cloud , I decided to go a different way an all-static comment handling system for Jekyll. First off, you need a way to render comments inside Jekyll. This was disturbingly easy with Jekyll s plugin system. With around 30 lines of Ruby, it was done. It simply reads a pile of YAML files from a _comments directory, and adds them to the individual blog post objects in the templating system, which can then be iterated over when rendering a post. Obviously, you need to also adjust your blog post templates to suit. This is fairly straightforward; just render each of page.comments in a list. Adding a comment submission form is also necessary, and similarly trivial. The only moving part in this whole system is the bit of code that handles comment form submissions. The simplest way to handle it, which is what I ve gone with, is a few lines of PHP that writes the contents of the form out as YAML and e-mails it to me. Although I m not a fan of PHP for web applications, this is sort of job is where PHP excels it s ubiquitous, trivial to deploy, and easy to understand (in small doses). Using a CGI or anything larger (like, say, a sinatra app) would be massive overkill. Since I m still a control freak at heart, I m hand-moderating all comments by the simple expedient of having all blog comments e-mailed to me¹, and I have to import them into the blog and re-upload. Mutt makes this trivial; I ve got a script that I trigger by pressing C (for comment) that pipes the message to formail to strip the RFC2822 header, then drops the rest into my blog s _comments directory. A quick rake upload and they re published. If you re not as obsessive as me about hand-moderating your comments, it wouldn t be hard to write a slightly different PHP script that commits the comment directly to a git repo² and re-renders the site. All you d have to do when you wanted to make a new post is merge those new comment commits from the live git repo and go to work. This would then be a completely static site (with all the performance benefits that implies) with immediate-gratification comments. So, if you re suspicious of the cloud, or just want to be the cool kid whose blog comments load even when people have turned cookies off, check out my Jekyll static comments plugin.

A side-bonus of running all my blog comments through my e-mail is that all of the comment spam gets handled through my existing spam filter, so there s no new complexities. The stuff that bogofilter is really sure is spam just goes straight to /dev/null, while the slightly-less-sure stuff goes into an IMAP folder I ignore except when I m extra-super-bored. Interestingly, bogofilter was already pre-trained and ready to go on my blog spam it caught most of the spam straight away, with no additional training from me, and after being trained on two examples, it s now going great guns auto-flushing the one persistent spammer who hammers one particular old post a couple of times a day with link farms.
I d strongly recommend having some sort of anti-spam setup before you blindly render the comments onto your site, lest your comments list looks like complete balls within a matter of days.

16 July 2011

Matthew Palmer: MySQL replication and crash recovery

The question was recently asked, How do I perform crash recovery on a MySQL master/slave database cluster? Because replication is async, a transaction that has been committed to the master may not be able to leave the master before the crash happens. The short answer is, you don t. Asynchronous replication, by it s very nature, is prone to these sorts of issues, and MySQL doesn t make life easier in general by using query replay. Some of the issues aren t massive: for instance, if a slave crashes, the master will keep replication logs for the slave until it comes back up (as long as the slave comes back before the master decides some of the replogs are too old and starts deleting them, of course), so as long as you can live without a master for a while, you can recover and continue along your merry way. Promoting a slave to be a master, on the other hand, is a situation frought with peril. Here, you ve just got to hope hope that your chosen slave is consistent with the master s state at death (because it s async, you have no guarantees about that), and that all the other slaves have the same ideas about what counts as the current state of the data. If your newly-promoted slave managed to apply an update that another slave didn t, that slave will be out of sync with reality (until such time as the now-dead master comes back and replays all those queries hahahaha). To even guarantee that any slaves have a consistent view of the data as compared to the new master, you ve got to rereplicate everything from the new master, because MySQL s where are you up to? counter is master-specific. I ve heard people who have to deal with this sort of thing these days say that the Maatkit tools are handy for dealing with various issues of this type (monitoring, repairing the database when it gets inconsistent). However, I prefer to keep right the hell away from MySQL replication altogether, after a year of having to gently coax an insane MySQL tree replication setup to do the right thing for any extended period, and watching a group of DBAs at Engine Yard go slowly insane. My preferred alternatives are:

For pure HA, replicate at the disk level using DRBD. You lose some of the supposed benefits of MySQL replication ( you can do dump backups off the slave! Woooooo! ), but you at least get consistency.
If you think you need a harem¹ for load purposes, use some other way of scaling MySQL that doesn t involve throwing an update to an amorphous blob of servers and hoping it sticks in the right places. Sharding can work, as can aggressive caching (intelligent storage of precalculated query results in memcached can work wonders) or just buying a bigger machine.
If you want to do anything fancy (my tree-replication scheme falls into this category), just don t. MySQL replication isn t the answer to these sorts of situations, it s the question. The answer is No! .

Never discount the value of the Jaws approach to scaling ( we re gonna need a bigger boat^Wmachine ) hardware is so ridiculously cheap, relative to the clueful sysadmin time needed to make (and keep) MySQL replication running, that you really need to be at the huge end of town (where other scaling issues are going to bite you in the arse first) before spending too much time on your database is worth the investment. Even then, you ll get more bang for your buck from:

Optimising your queries;
Optimising your MySQL configuration;
Using a k/v store for really non-relational data (sessions, I m looking at you)
Sharding your data

than you ever will from the constant, mind-numbing grind that is keeping MySQL replication going.

The term comes from a time when a colleague near-screamed This slave cluster keeps fucking me! at a particularly intransigent set of MySQL slaves.

14 July 2011

Matthew Palmer: More Nginx SSL Performance Tips

Well, actually, just one, and it s not even mine. Matt Stancliff benchmarked a few different methods of SSL termination and found that nginx sucked rocks relative to the other options. However, it turns out the comparison wasn t as balanced as he originally thought, and discovered that the SSL ciphers you use are quite important. As an aside, I like Matt s benchmark everything philosophy.

13 July 2011

Matthew Palmer: "Wireless Internet is the future" my arse

Here in Australia we re building out a giant, nationwide fiber network (with degraded service to the few parts of the country that aren t heavily populated like, nearly all of it), known as the National Broadband Network . Many critics of the planned network are saying that it s technology is outdated, and that the future of last-mile Internet access is wireless; that by the time the network is completed, we ll have no need for fibre optic cable to the home, and we ll all just use some magical wireless network for all our Internetting needs. Well, right at this moment, I m sitting in the middle of Australia s largest city, with full bars of coverage being reported by my phone, and getting ping times of over 3 minutes (you heard that right; my record so far is ping reporting time=194001 ms for a reply) and no ability to usefully access the Internet. If wireless is the future, I hope it s a long way away. To misquote a great man, I WANT MY ADSL BACK YOU SON OF A BITCH!

7 July 2011

Matthew Palmer: Honest Achmed is my CA of Choice

It s not often that you get clarity and transparency from a Certificate Authority, so it s good to see that Honest Achmed has requested his CA root certificate be included in Firefox. Really, how can you not trust a guy whose name is Honest Achmed , and who has such a strong and clear-cut verification statement:

Honest Achmed promises to studiously verify that payment from anyone requesting a certificate clears before issuing it (except for his uncles, who are good for credit). Achmed guarantees that no certificate will be issued without payment having been received, as per the old latin proverb nil certificati sine lucre .

His response to Mozilla s CA Recommended Practices is forthright and trustworthy, too:

Honest Achmed promises to abide by these practices. If he s found not to abide by them, he ll claim it was a one-off slip-up in procedures and that policies have been changed to ensure that it doesn t happen again. If it does happen again, he ll blame it on one of his uncles or maybe his cousin, who still owes him some money for getting the car fixed.

I m getting all my SSL certificates from Honest Achmed. He can t be any worse than Comodo

1 July 2011

Matthew Palmer: Metablogging

I ve taken the plunge, and converted my blog from a venerable but largely unmaintain(ed able) blosxom setup that I ve had since 2005 (OMG!) into a new and shiny Jekyll-backed system. The main benefits are that I m more comfortable hacking away at Jekyll (being written in Ruby), and my blog post submission no longer requires me to e-mail my blog posts to myself to put them up (that was just an artifact of how I set it up all those years ago, but it had worked so I wasn t willing to change it just for the sake of it). Jekyll s superior (for me) hackability has reaped dividends already I ve put together a home-grown comments system, which fulfils my primary requirements of being as close to completely statically-generated as it s possible to be (about 10 lines of PHP is the only dynamic content), and doesn t involve JavaScript or putting my blog comments in the cloud .

28 June 2011

Matthew Palmer: Water Saving Shower Heads

In a series of experiments performed over the past few days, I've determined that the actual mechanism by which "water saving" shower heads work is not, as you might naively imagine, reducing the amount of water dispersed per unit time. Instead, they make showering such an unpleasant experience that you perform amazing feats of speed washing so you can get out as quickly as possible. In extreme cases, I can imagine you would save even more water by just not washing at all.

Matthew Palmer: Water Saving Shower Heads

Matthew Palmer: SSL Session Caching (in nginx)

Given that I was responsible for putting together the system architecture for Github when they moved away from their previous services provider to Anchor Systems, I keep an eye on what's going on there even though I've gotten out of professional IT. Recently(ish), Ryan Tomayko blogged about some experiments in using Amazon CloudFront as a CDN for static assets. While discussing as-yet unsolved performance issues, he quoted a tweet from Shopify's CEO (who I would have thought would have known better):

Protip to make SSL fast: terminate in your load balancer, that makes SSL session cookies actually work.

	gnutls-cli -V -r HOSTNAME  grep 'Session ID'

If all three Session IDs are the same, then you've got SSL session caching running. (If you're wondering why I'm using gnutls-cli rather than openssl s_client, it's because openssl is living in the pre-IPv6 dark ages, and I've got IPv6-only services running I wanted to test). So that's great for your little single-server hosting setup, but per-machine caches aren't much use when you're running behind a load balancer, Github style, because each SSL-enabled TCP connection can very well end up on a different server, and a per-machine cache won't help you very much. (If you think the answer to this problem is "session affinity", you're setting yourself up for a whole other world of pain and suffering.) Here's the kicker, though -- the SSL session is just a small, opaque chunk of bytes (OpenSSL appears to store it internally as an ASN.1 string, but that's an irrelevant implementation detail). An SSL session cache is just a mapping of the SSL session ID (a string) to the SSL session data (another string). We know how to do "shared key-value caches" -- we have the technology. There won't be too many large-scale sites out there that aren't using memcached (or something practically equivalent). So, rather than stuff around trying to run a load balancer with enough grunt to handle all your SSL communication for all time, you can continue to horizontally scale the whole site with as many backend webservers as you need. All that's needed is to teach your SSL-using server to talk to a shared key-value cache. Far be it for me to claim any sort of originality in this idea, though -- patches for Apache's mod_ssl to do this have been floating around for a while, and according to the release notes, the 2.4 series will have it all built-in. However, Github uses nginx, to great effect, and nobody's done the work to put this feature into nginx (that I could find, anyway). Until now. I present, for the edification of all, a series of patches to nginx 0.8.x to provide memcached-enabled SSL session caching. Porting them to other versions of nginx hopefully won't be too arduous, and if you ask nicely I might give it a shot. If you don't use memcached (gasp!), my work should at least be a template to allow you to hook into whatever system you prefer. Sharding the memcached calls wouldn't be hard either, but I'm not a huge fan of that when other options are available. I had originally hoped to be able to say "it works at Github!" as a testimonial, but for some reason it hasn't been deployed there yet (go forth and pester!), so instead I'm just releasing it out there for people to try out on a "well, I don't know that it doesn't work" basis. If it breaks anything, it's not my fault; feel free to supply patches if you want to fix anything I've missed.

27 June 2011

Matthew Palmer: SSL Session Caching (in nginx)

Protip to make SSL fast: terminate in your load balancer, that makes SSL session cookies actually work.

No, no, NO -- a million times NOOOOOO! Load balancers are, by their very nature, performance chokepoints. You can't horizontally scale your load balancer, because if you do (by, say, putting another load balancer in front of it -- don't laugh, I've seen it done) you've just re-introduced all the same problems that putting SSL on the load balancer was supposed to accomplish (that is, getting SSL session caching to work). The only thing you accomplish by putting your SSL sessions onto the load balancer is bring the moment your load balancer melts down through excessive CPU usage that much closer. I dunno, maybe Shopify doesn't actually do that much traffic or something, but I certainly wouldn't be signing up for that headache if it were my infrastructure. (Sidenote: yes, I know that you can get "SSL hardware accelerators", that put the basic crypto operations into dedicated silicon, but they're rarely cheap, and you can't horizontally scale those, either. They're effectively a specialised and outrageously expensive form of "scaling by bigger hardware" -- which is a valid method of scaling, but one with fairly terminal limitations that certainly wouldn't make it practical for Github). The problem you're trying to solve by centralising SSL operations is that SSL includes a mechanism whereby a client making a new TCP connection can say "here's the crypto details from last time I connected to you" and the server can just pull out all of the per-connection data that would otherwise take a long time (several TCP round trips and a smallish chunk of CPU time) to setup. This can make SSL connections a lot snappier (in Ryan's opinion, "We spend roughly as much time handshaking as we do generating the average dynamic request."), so they're a definite win if you can do it (and yes, you can do it). Webservers don't typically do this caching by default, but setting up a simple SSL session cache for a single nginx or Apache webserver is pretty trivial. Testing it is simple, too: just install gnutls-bin and run:gnutls-cli -V -r HOSTNAME grep 'Session ID' If all three Session IDs are the same, then you've got SSL session caching running. (If you're wondering why I'm using gnutls-cli rather than openssl s_client, it's because openssl is living in the pre-IPv6 dark ages, and I've got IPv6-only services running I wanted to test). So that's great for your little single-server hosting setup, but per-machine caches aren't much use when you're running behind a load balancer, Github style, because each SSL-enabled TCP connection can very well end up on a different server, and a per-machine cache won't help you very much. (If you think the answer to this problem is "session affinity", you're setting yourself up for a whole other world of pain and suffering.) Here's the kicker, though -- the SSL session is just a small, opaque chunk of bytes (OpenSSL appears to store it internally as an ASN.1 string, but that's an irrelevant implementation detail). An SSL session cache is just a mapping of the SSL session ID (a string) to the SSL session data (another string). We know how to do "shared key-value caches" -- we have the technology. There won't be too many large-scale sites out there that aren't using memcached (or something practically equivalent). So, rather than stuff around trying to run a load balancer with enough grunt to handle all your SSL communication for all time, you can continue to horizontally scale the whole site with as many backend webservers as you need. All that's needed is to teach your SSL-using server to talk to a shared key-value cache. Far be it for me to claim any sort of originality in this idea, though -- patches for Apache's mod_ssl to do this have been floating around for a while, and according to the release notes, the 2.4 series will have it all built-in. However, Github uses nginx, to great effect, and nobody's done the work to put this feature into nginx (that I could find, anyway). Until now. I present, for the edification of all, a series of patches to nginx 0.8.x to provide memcached-enabled SSL session caching. Porting them to other versions of nginx hopefully won't be too arduous, and if you ask nicely I might give it a shot. If you don't use memcached (gasp!), my work should at least be a template to allow you to hook into whatever system you prefer. Sharding the memcached calls wouldn't be hard either, but I'm not a huge fan of that when other options are available. I had originally hoped to be able to say "it works at Github!" as a testimonial, but for some reason it hasn't been deployed there yet (go forth and pester!), so instead I'm just releasing it out there for people to try out on a "well, I don't know that it doesn't work" basis. If it breaks anything, it's not my fault; feel free to supply patches if you want to fix anything I've missed.

13 June 2011

Raphaël Hertzog: Why you should always have a network connection when installing Debian

This is a simple tip but an important one: when you re installing Debian, take the time required to ensure the machine is connected to the Internet with a wired connection. If you have DHCP available, the debian-installer will use it to configure the network.

Why not use the wireless connection? Because debian-installer in Squeeze doesn t support WPA encryption, but only WEP. So if you re using WPA, picking the wireless connection will lead to no working network during the installation and this is to be avoided. If you re still using WEP, you can go ahead of course. If you only have a wireless connection with WPA, your might want to help the debian-installer team and add the required support. Matthew Palmer did some work on it a few months ago (see this mail and his branch in the netcfg git repository) but he resigned from the d-i team in the mean time. So WPA support is still not available in the wheezy debian-installer. Why is the network so important?

The tasks that you select during the installation process might suggest installation of supplementary packages that are not available on your installation disc. If you install without network, the resulting system might differ from the expected one since it will be missing some packages that are available in the Debian repositories but not on your installation disc.
Your installation media might be old and there are security updates that have been published. If you do your initial installation with network, the security updates will be installed before the reboot and thus before the services are exposed over the network.
If you re not installing a desktop with network-manager (Debian s default GNOME Desktop provides it), the initial network configuration is important since this configuration is kept for the future. And you surely want network connectivity on your machine, don t you?
Without network, APT s sources.list will not be properly configured to include an HTTP mirror of your country. And really, I prefer when apt-get install can work without the initial installation disc.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

22 comments Liked this article? Click here. My blog is Flattr-enabled.

Next.

Previous.